Evaluation of ChatGPT 4.0 in Thoracic Imaging and Diagnostics

Area of Science:

• Artificial Intelligence in Medicine
• Natural Language Processing Applications
• Thoracic Radiology Diagnostics

Background:

• Natural Language Processing (NLP) models, including ChatGPT 4.0, are increasingly impacting medical data analysis and clinical workflows.
• The diagnostic capabilities of advanced NLP tools in specialized medical fields like thoracic imaging require thorough evaluation.

Purpose of the Study:

• To assess the diagnostic performance of ChatGPT 4.0 in answering multiple-choice questions related to thoracic imaging.
• To analyze the accuracy and variability of ChatGPT 4.0's diagnostic capabilities across different thoracic imaging subgroups.

Main Methods:

• Utilized ChatGPT 4.0 to answer multiple-choice questions derived from thoracic imaging scenarios.
• Conducted rigorous statistical analysis to evaluate accuracy and identify performance variations across subgroups.

Main Results:

• Achieved an overall diagnostic accuracy of 84.9% for thoracic radiology questions.
• Demonstrated perfect scores in terminology and diagnostic signs, high accuracy in intensive care (90%) and normal anatomy (80%).
• Pathology subgroup accuracy averaged 89.1% (excelling in infectious pneumonia, atelectasis), while disease-related questions averaged 79.1% (lower in vascular disease at 50% and lung cancer at 66.7%).

Conclusions:

• ChatGPT 4.0 shows significant potential for assisting in thoracic condition diagnosis.
• Identified variability in performance necessitates further research and refinement of AI transformer architectures for enhanced clinical reliability.