Enhancing Accessibility of Medical Texts through Large Language Model-Driven Plain Language Adaptation
Ting-Wei Chang, Hen-Hsen Huang, Hsin-Hsi Chen
cs.CL
Quick verdict
Broad PLABA-2024 study of prompting, QLoRA and MoA for medical text simplification; solid empirical sweep, but MoA gains are unsupported and stats absent.
Summary
This paper tackles Plain Language Adaptation (PLA) — rewriting complex biomedical abstracts for lay readers — in the context of the PLABA/TREC 2024 shared task. The authors compare zero-shot prompting, random few-shot in-context learning (ICL), and similarity-based example selection across Gemini, GPT-4o-mini, LLaMA, Gemma-2 and Mistral families; apply QLoRA finetuning (rank/alpha 16, one epoch); and test a single-layer Mixture-of-Agents (MoA) setup with Gemini-1.5-flash as aggregator. Similarity-based few-shot selection improves relevance metrics monotonically with shot count (e.g., Gemini-1.5-flash BLEU 14.15→21.37 from zero- to 5-shot). Finetuning raises n-gram overlap and factuality scores substantially, yet SARI falls for most models and FKGL readability worsens, tempering the paper's 'comprehensive improvement' claim; MoA shows negligible differences in automatic metrics. In the official TREC 2024 human evaluation their best system ranked 4th among listed PLA submissions (with finetuning) and 8th without, and 2nd in term-replacement identification. Limitations include reliance on the single PLABA dataset, absence of statistical significance testing, unpublished prompts and code, no evaluation of replacement generation, and a Gemini-family LLM judge that may favor Gemini outputs.
Review complete
Assessment at a glance
6
risk flags identified
● high 0
● medium 3
● low 3
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Few-shot ICL with semantically similar examples improves relevance metrics monotonically as the number of shots grows, and outperforms random example selection.
- QLoRA finetuning improves n-gram relevance and factuality scores across model families, which the paper frames as comprehensive improvement despite SARI declines and FKGL increases in Table 3.
- Integrating a single-layer Mixture-of-Agents setup (Gemini-1.5-flash aggregator, seven contributor models) was expected to enhance robustness; Table 4 shows near-identical automatic metrics with and without it.
- In the official TREC 2024 human evaluation, the team's best PLA system ranked 4th with finetuned models and 8th without, and the team placed 2nd in term-replacement identification.
Strengths
Broad, controlled empirical sweep across model families and prompting regimes
Table 2 ablates shot count 0-5 for random vs. similarity-selected ICL on three API models; Table 3 covers nine open models (LLaMA-2/3/3.1, Gemma-2, Mistral) with and without QLoRA — an informative map for practitioners.
Clear monotonic few-shot effect with a actionable selection strategy
Sec. 4.2.2 / Table 2: similarity-based example selection consistently beats random selection and improves with shots (Gemini-1.5-flash BLEU 14.15→21.37; SARI 43.21→46.04 from zero- to 5-shot).
Official external human evaluation, honestly reported
Multi-dimensional evaluation protocol aligned with shared-task norms
Candid scope limitations
Weaknesses
Headline claim contradicted by the paper's own tables
MoA is marketed as a key contribution but shows a null result
No inferential statistics anywhere
Reproducibility gaps
Internal inconsistencies undermine confidence in the setup
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper reports the NTU NLG team's systems (team ntu_nlp) for the TREC 2024 PLABA track: (1) the Plain Language Adaptation (PLA) task — rewriting biomedical abstracts sentence-by-sentence for lay readers — and (2) the new Term Replacement task. Five contributions are claimed (Sec. 1): a comparison of zero-shot vs. random vs. semantic-similarity few-shot ICL; QLoRA finetuning of open LLMs; integration of Mixture-of-Agents (MoA); an automated-plus-LLM-judge evaluation framework; and evaluation on the term replacement task. The headline claim — that LLM-driven PLA is effective while 'preserving essential content' — is only partially supported by the paper's own tables.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 6 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is Plain Language Adaptation (PLA) of biomedical texts?
PLA is the task of automatically rewriting technical biomedical writing, such as PubMed abstracts, into plain language that patients and the general public can understand while keeping the factual content intact. The PLABA shared task (run at TREC 2024) provides expert-written plain-language references to benchmark systems.
How well did this system do in the PLABA 2024 shared task?
In the official TREC 2024 human evaluation, the team's best PLA system averaged 0.8238 and ranked 4th among the listed submissions when using finetuned models (8th without finetuning); the top submission reached 0.8853. In the term replacement task, the team placed 2nd in identifying difficult terms.
Does finetuning make LLM output on medical texts simpler and better?
It is a trade-off in this paper. Finetuning raised overlap-based relevance metrics and factuality scores substantially, but the paper's own Table 3 shows SARI (a simplification metric) falling and FKGL grade level rising for most models — outputs matched the expert references more closely but were not easier to read.
Is code or data available for this paper?
The paper does not announce any code, prompt, or model release. The underlying PLABA dataset is publicly available from the U.S. National Library of Medicine, so others could rebuild similar systems, but exact replication of this paper's runs is not possible from the paper alone.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1-3 are not analyzable from the extracted text; Figure 3 contains the term-replacement test-set results.
- Tables were extracted with jumbled formatting; individual numbers could contain parsing artifacts, though cross-row checks are internally consistent.
- The text may be truncated; prompt templates or an appendix could be missing, which affects the reproducibility judgment.
- Some Table 6 entries are unavailable in the paper itself (organizers' incomplete judging), not an extraction failure.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.