In-Context Learning with Large Language Models: A Simple and Effective Approach to Improve Radiology Report Labeling

Area of Science:

• Artificial Intelligence in Medical Imaging
• Natural Language Processing in Healthcare
• Radiology Report Analysis

Background:

• Automated labeling of radiology reports is crucial for efficient data analysis and clinical insights.
• Generative Pre-trained Transformer-4 (GPT-4) shows potential for complex natural language understanding tasks.

Purpose of the Study:

• To evaluate the effectiveness of in-context learning with GPT-4 for labeling radiology reports.
• To compare standard prompts with context-aware prompts for improved labeling accuracy.

Main Methods:

• A retrospective study using radiology reports from the MIMIC-III database.
• Comparison of basic and in-context prompts for multilabel classification on head and abdominal CT reports.
• Optimization experiments to assess consistency and error rates.

Main Results:

• In-context learning with GPT-4 significantly improved F1-scores for labeling head CT reports, particularly for 'foreign body' and 'mass' labels.
• Substantial performance gains were observed across all labels for abdominal CT reports using in-context prompts.
• Inter-reader accuracies were high, indicating reliable baseline performance.

Conclusions:

• In-context learning using GPT-4 consistently enhances radiology report labeling accuracy.
• This approach effectively aligns model criteria with human annotators for both subjective and objective labeling tasks.
• The method is practical, adaptable, and suitable for diverse labeling applications in radiology research.