Summarizing, Simplifying, and Synthesizing Medical Evidence Using GPT-3 (with Varying Success).

Area of Science:

• Biomedical Informatics
• Artificial Intelligence
• Natural Language Processing

Background:

• Large language models (LLMs) demonstrate proficiency in summarizing general domain texts.
• The efficacy of LLMs in specialized, high-stakes fields like biomedicine remains largely unexplored.
• Evaluating LLM performance in biomedical text summarization requires domain-specific expertise.

Purpose of the Study:

• To assess the capability of GPT-3 in generating high-quality summaries of biomedical research articles.
• To evaluate GPT-3's performance in both single-document and multi-document summarization settings.
• To determine the factual accuracy and synthesis capabilities of GPT-3 in a biomedical context.

Main Methods:

• Domain experts with medical training evaluated GPT-3 generated summaries.
• Summaries were generated in zero-shot settings for single and multiple biomedical articles.
• An annotation scheme focused on factual accuracy was developed for evaluation.

Main Results:

• GPT-3 accurately summarizes and simplifies single biomedical articles, including randomized controlled trials.
• GPT-3 exhibits significant challenges in synthesizing and accurately aggregating findings across multiple biomedical documents.
• Factual accuracy was a key consideration in the expert evaluation of generated summaries.

Conclusions:

• While capable of single-document summarization, GPT-3's multi-document synthesis in biomedicine requires improvement.
• Expert evaluation is crucial for assessing the reliability of LLM-generated biomedical summaries.
• Further research is needed to enhance LLM performance in complex biomedical evidence synthesis.