PlainQAFact: Retrieval-augmented factual consistency evaluation metric for biomedical plain language summarization

Area of Science:

• Natural Language Processing
• Medical Communication
• Artificial Intelligence

Background:

• Large language models (LLMs) can generate inaccurate information, posing risks in healthcare, especially for the public.
• Current methods for checking factual consistency in simplified medical texts struggle with added explanations.
• Elaborative explanations, while aiding comprehension, complicate factual verification.

Purpose of the Study:

• To develop an automatic metric for evaluating factual consistency in plain language summaries (PLS) of medical information.
• To address the limitations of existing methods in handling elaborative explanations within PLS.
• To introduce a new benchmark and tool for reliable medical communication.

Main Methods:

• Introduced PlainQAFact, a novel metric trained on the PlainFact dataset.
• Developed a two-step process: sentence type classification followed by retrieval-augmented QA scoring.
• Evaluated PlainQAFact against existing metrics on source-simplified and elaborately explained sentences.

Main Results:

• Existing evaluation metrics demonstrated significant failures in assessing factual consistency for PLS, particularly with elaborative content.
• PlainQAFact consistently outperformed all existing metrics across various evaluation scenarios.
• Analysis refined the metric's effectiveness concerning external knowledge, answer extraction, and document granularity.

Conclusions:

• PlainQAFact offers a robust solution for evaluating factual consistency in biomedical PLS, especially for complex explanations.
• The metric is crucial for ensuring the accuracy and safety of health information disseminated to the public.
• The study provides a valuable tool and benchmark for advancing trustworthy plain language communication in medicine.