Lessons from the TREC Plain Language Adaptation of Biomedical Abstracts (PLABA) track.

Area of Science:

• Biomedical Informatics
• Natural Language Processing
• Health Communication

Background:

• Advances in Large Language Models (LLMs) offer potential for translating complex biomedical literature into accessible plain language for patients and caregivers.
• The inherent unpredictability and potential risks associated with LLMs in this sensitive domain necessitate rigorous evaluation frameworks.

Purpose of the Study:

• To stimulate research in adapting biomedical literature to plain language.
• To provide high-quality evaluation of LLM systems for this task.

Main Methods:

• The Plain Language Adaptation of Biomedical Abstracts (PLABA) track at Text Retrieval Conferences (TRC) 2023-2024.
• Task 1: Sentence-level rewriting of 400 abstracts; Task 2: Identifying and replacing difficult terms in 300 abstracts.
• Utilized a four-fold professionally-written reference set for automatic evaluation and extensive manual evaluation by biomedical experts.

Main Results:

• Top-performing models in Task 1 demonstrated human-level factual accuracy and completeness but lacked simplicity and brevity.
• Automatic evaluation metrics showed poor correlation with manual judgments.
• Systems struggled with term identification in Task 2 but excelled in accuracy, completeness, and simplicity for replacements, though brevity remained a challenge.

Conclusions:

• The PLABA track highlighted the potential of LLMs for biomedical text simplification.
• LLMs still exhibit deficiencies, underscoring the need for better automatic benchmarking tools for plain language adaptation.