Perspectives on Cross-Lingual Consistency in LLMs for Medical Questions
Minh Duc Bui, Mario Sanz-Guerrero, Abteen Ebrahimi, Sagi Shaier, Peter Herbert Kann, Manuel Mager, Katharina von der Wense
cs.CL
Quick verdict
Stakeholders split on cross-lingual consistency in medical LLMs; useful first evidence, though thin statistics and persona averaging temper conclusions.
Summary
The paper synthesizes multilingual medical NLP into two stances — consistency (correct answers are language-invariant) versus cultural adaptation — and identifies three gaps: stakeholder absence, no user-outcome evidence, and no benchmarks separating universally correct from culture-specific items. It then runs two studies. First, a forced-choice survey of 348 professionals (170 medical, 121 NLP, 57 anthropology) in Germany, Spain, and the US: anthropologists favor adaptation in every country (80–82%); NLP respondents hover near 60% consistency; medical respondents split (65% consistency in Germany, significant vs. chance; 43% in the US, with the US–Germany difference significant by z-test). A PubMed MeSH analysis mirrors this ranking in culture-related publication share (US 0.46%, Spain 0.37%, Germany 0.29%). Second, 11 open-weight LLMs answer the survey under profession and country personas (3 languages, n=20 per condition): models overestimate consistency support among NLP (80.0% vs. 61.2%) and medical (83.6% vs. 53.7%) personas and show no country sensitivity. Main limitations: the binary instrument conflates content and communication adaptation; the sample is Global-North-only; no code or data release is stated; statistical reporting lacks confidence intervals, effect sizes, and multiple-comparison correction; and aggregate LLM results average over several near-deterministic models.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 2
● low 3
5 strengths · 5 weaknesses · 5 author questions
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Multilingual medical NLP splits into two stances — consistency (language-invariant correct answers) and cultural adaptation — and the field lacks stakeholder input, user-outcome evidence, and benchmarks separating universal from culture-specific items (Section 3.3).
- Anthropologists consistently favor adaptation (80–82% in all three countries, significant vs. chance), while medical and NLP professionals show no significant consensus in 5 of 6 profession-country cells (Figure 2, Section 4.4).
- US medical professionals lean toward adaptation (57%) whereas German and Spanish counterparts lean toward consistency, with the US–Germany difference significant by a two-sided proportion z-test (Section 4.4).
- LLMs prompted with profession and country personas overestimate support for consistency among NLP (80.0% vs. 61.2%) and medical (83.6% vs. 53.7%) personas and fail to reproduce the cross-national variation (Table 2, Figure 4a).
- Neither consistency nor adaptation can currently be considered clearly preferable; persona-prompted steering alone is unlikely to be a sufficient adaptation mechanism (Sections 4.5, 5.3).
Strengths
First joint multi-stakeholder survey on this exact question
Section 4 surveys 348 professionals across medicine (n=170), NLP (n=121), and anthropology (n=57) in three countries with three-stage profession filtering (Prolific screeners, in-survey domain check, verification questions), directly closing the stakeholder gap the authors identify in Section 3.3.
Balanced two-stance literature synthesis with actionable gaps
Section 3 represents both camps via their strongest works (XLingHealth's 'Better to Ask in English' vs. CMB/TCM, Dey et al.'s HealthBench misalignment, Africa Health Check) and derives three concrete gaps, including the item-level universal-vs-culture-specific distinction, which is the sharpest formulation of this problem in the literature reviewed.
Useful negative result on persona simulation
Independent triangulation of the US–Germany medical divergence
Transparent ethics and limitations
Weaknesses
Statistical reporting is below bar for the claims made
Aggregate LLM result conceals heavy model degeneracy
Binary instrument conflates the two adaptation axes the paper itself distinguishes
Sampling asymmetry and channel confounding across countries
Opinion evidence cannot touch the paper's own gap #2
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Preliminary Review: Perspectives on Cross-Lingual Consistency in LLMs for Medical Questions
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
Should medical LLMs give the same answer regardless of the language of the question?
This paper argues the question is unresolved. Its survey of 348 professionals in Germany, Spain, and the US found no significant consensus among medical and NLP respondents in most groups — only anthropologists clearly favored culturally adapted answers (80–82% in every country) — and the authors conclude that neither cross-lingual consistency nor cultural adaptation is currently demonstrably better for users.
Can LLM personas replace human expert surveys?
Not according to this study. Eleven open-weight models prompted with profession and country personas overestimated support for language-invariant answers among NLP personas (80.0% versus 61.2% for humans) and medical personas (83.6% versus 53.7%), and failed to reproduce the country-level differences observed in human respondents, making them unreliable proxies for stakeholder opinion.
Who was surveyed and what were the key numbers?
348 professionals across three countries: 170 medical, 121 NLP, and 57 anthropology respondents in Germany, Spain, and the US. Anthropologists favored adaptation everywhere; German medical respondents were the only group significantly favoring consistency (65%), US medical respondents leaned toward adaptation (57%), and the US–Germany difference was statistically significant.
Why might medical answers legitimately differ across languages or countries?
The paper gives concrete examples: clinical guidelines differ by region (US versus German-speaking osteoporosis risk tools use different variable sets), traditional medicine paradigms such as Traditional Chinese Medicine fall outside Western benchmarks, and culturally grounded advice — e.g., India-specific dietary guidance — has been scored as incorrect by US-centric rubrics like HealthBench despite being locally appropriate.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figure 2's bar labels are extracted in garbled order; all percentages cited in this review were cross-checked against the prose of Section 4.4 and Figure 4 rather than taken from the garbled figure text.
- The provided metadata abstract states 356 participants whereas the full-text abstract, Section 4.2, and Table 1 all state 348; this review uses 348 and flags the discrepancy rather than treating it as an extraction artifact.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.