Comparing Commercial and Open-Source Large Language Models for Labeling Chest Radiograph Reports

Area of Science:

• Artificial Intelligence in Medical Imaging
• Natural Language Processing for Radiology
• Machine Learning in Healthcare

Background:

• Large language models (LLMs) are rapidly advancing, with numerous commercial and open-source options available.
• Previous studies focused on GPT-4 for radiology report analysis, but real-world comparisons with leading open-source LLMs are lacking.
• Accurate extraction of findings from chest radiograph reports is crucial for clinical decision-making.

Purpose of the Study:

• To compare the performance of leading open-source LLMs against GPT-4 for extracting relevant findings from chest radiograph reports.
• To evaluate the effectiveness of zero-shot and few-shot prompting strategies in this task.

Main Methods:

• Retrospective analysis of two independent datasets of free-text chest radiograph reports (ImaGenome and Massachusetts General Hospital).
• Comparison of commercial models (GPT-3.5 Turbo, GPT-4) with open-source models (Mistral-7B, Mixtral-8×7B, Llama 2-13B, Llama 2-70B, Qwen1.5-72B) and CheXbert/CheXpert-labeler.
• Evaluation using zero-shot and few-shot prompting, with performance measured by F1 scores and compared using the McNemar test.

Main Results:

• On the ImaGenome dataset, Llama 2-70B achieved micro F1 scores of 0.97 (zero-shot) and 0.97 (few-shot), closely matching GPT-4's 0.98.
• On the institutional dataset, an ensemble open-source model achieved micro F1 scores of 0.96 (zero-shot) and 0.97 (few-shot), comparable to GPT-4's 0.98 and 0.97.
• GPT-4 demonstrated superiority in zero-shot labeling, but few-shot prompting significantly improved open-source model performance, yielding comparable results.

Conclusions:

• While GPT-4 excelled in zero-shot report labeling, few-shot prompting with minimal examples enabled open-source LLMs to achieve performance levels close to GPT-4.
• The effectiveness of few-shot prompting varied across different datasets and LLM architectures.
• Open-source LLMs show significant potential for clinical applications in radiology report analysis, especially when fine-tuned with few-shot learning.