Using Large Language Models to Summarize Evidence in Biomedical Articles: Exploratory Comparison Between AI- and...

Area of Science:

• Artificial Intelligence
• Natural Language Processing
• Scientific Communication

Background:

• Annotated bibliographies are crucial for literature review but require significant human expertise and time.
• AI tools offer potential for efficient summarization, yet may introduce errors.

Purpose of the Study:

• To evaluate the feasibility of using ChatGPT for generating annotated bibliographies.
• To compare AI-generated annotations with human-written ones regarding accuracy, conciseness, and quality.

Main Methods:

• Two human annotators and three versions of ChatGPT (3.5, 4, 5) independently annotated 15 publications.
• Annotations were assessed for word count, readability (Flesch Reading Ease), capture of main points, presence of errors, and inclusion of quality/context.
• Statistical models were used to compare annotations generated by humans and AI.

Main Results:

• ChatGPT annotations were longer and less readable than human annotations.
• No significant difference was found in capturing main points between AI and human annotations.
• Human annotations had fewer errors, while AI annotations more consistently included quality and context, though sometimes inaccurately.

Conclusions:

• AI tools like ChatGPT can efficiently generate literature summaries, providing consistent quality and context.
• AI-generated annotations may contain more errors and inaccuracies, necessitating human verification.
• Further research into prompt engineering is needed to enhance AI chatbot performance for scientific literature summarization.