Evaluating large language models for information extraction from gastroscopy and colonoscopy reports through...

Area of Science:

• Medical Informatics
• Artificial Intelligence in Medicine
• Natural Language Processing

Background:

• Automated analysis of clinical text is crucial for improving healthcare efficiency.
• Large language models (LLMs) offer potential for extracting information from unstructured medical reports.
• Endoscopic procedures generate extensive reports requiring efficient summarization and analysis.

Purpose of the Study:

• To systematically evaluate large language models (LLMs) for automated information extraction from gastroscopy and colonoscopy reports.
• To assess LLMs' capabilities in structured data extraction, pattern recognition, and diagnostic support using prompt engineering.
• To understand the performance of various LLMs under different learning paradigms and prompting strategies.

Main Methods:

• Developed a three-task evaluation framework: entity extraction, pattern recognition, and diagnostic assessment.
• Utilized a dataset of 162 expert-annotated endoscopic reports.
• Evaluated multiple LLMs (proprietary, emerging, open-source) using zero-shot and few-shot learning with direct and Chain-of-Thought (CoT) prompting.

Main Results:

• Larger, specialized LLMs excelled in basic entity extraction but struggled with spatial relationships and integrating clinical findings.
• Few-shot learning effectiveness varied, with larger models demonstrating more consistent performance gains.
• Prompt engineering strategies, including CoT variants, influenced model performance across tasks.

Conclusions:

• LLMs show potential for analyzing specialized medical texts like endoscopy reports.
• Current LLMs have limitations in capturing complex clinical nuances and spatial information.
• Findings inform the development of advanced clinical documentation analysis systems using AI.