Enhancing Clinical Data Extraction from Pathology Reports: A Comparative Analysis of Large Language Models

Area of Science:

• Natural Language Processing
• Medical Informatics
• Artificial Intelligence in Healthcare

Background:

• Pathology reports contain complex, narrative-based clinical data.
• Manual extraction of critical information is time-consuming and prone to errors.
• Small large language models (sLLMs) offer potential for automated data extraction.

Purpose of the Study:

• To evaluate the efficacy of sLLMs in extracting information from multi-center free-text pathology reports.
• To compare the performance of different Llama 2 model sizes (7B, 13B, 70B parameters).
• To assess the impact of zero-shot versus five-shot learning on model accuracy.

Main Methods:

• Utilized three variants of the Llama 2 model (7B, 13B, 70B parameters).
• Evaluated performance in zero-shot and five-shot learning settings.
• Employed a regular expression-based information extraction tool as a benchmark.

Main Results:

• Significant performance variations observed across different model sizes and learning settings.
• The 70 billion parameter Llama 2 model demonstrated superior accuracy in the five-shot scenario.
• Example-driven learning (five-shot) significantly improved model performance.

Conclusions:

• sLLMs, particularly larger variants like the 70B Llama 2 model, show strong potential for accurate pathology report data extraction.
• Five-shot learning enhances the performance of sLLMs for clinical text analysis.
• Integrating sLLMs can improve efficiency and accuracy in clinical data extraction, benefiting patient care and research.