ChatGPT vs. Gemini: Comparative accuracy and efficiency in Lung-RADS score assignment from radiology reports

Area of Science:

• Artificial Intelligence in Radiology
• Medical Imaging Analysis
• Lung Cancer Screening

Background:

• Lung-RADS (Lung Imaging Reporting and Data System) is crucial for standardizing lung cancer screening.
• Accurate Lung-RADS scoring relies on interpreting low-dose computed tomography (LDCT) reports.
• Large Language Models (LLMs) show potential for automating medical report analysis.

Purpose of the Study:

• To assess the accuracy of various LLMs in generating Lung-RADS scores.
• To compare the performance of ChatGPT-3.5, ChatGPT-4o, Google Gemini, and Gemini Advanced.
• To evaluate LLM efficiency in processing LDCT radiology reports.

Main Methods:

• Retrospective analysis of 242 LDCT radiology reports.
• LLMs assigned Lung-RADS scores based on report findings without fine-tuning.
• Accuracy compared against radiologist-assigned scores; response times measured.

Main Results:

• ChatGPT-4o achieved the highest accuracy (83.6%) in Lung-RADS scoring.
• ChatGPT-3.5 (70.1%), Gemini (70.9%), and Gemini Advanced (65.1%) showed lower accuracy.
• ChatGPT-4o had the greatest agreement with radiologists (κ = 0.836), but below human interobserver levels.

Conclusions:

• ChatGPT-4o surpassed other LLMs in Lung-RADS score accuracy but did not match human expert performance.
• LLMs show potential for lung cancer screening report analysis.
• Domain-specific training is necessary to enhance LLM reliability for clinical decision-making.