Evaluation of GPT-4's Chest X-Ray Impression Generation: A Reader Study on Performance and Perception

Area of Science:

• Artificial Intelligence in Medicine
• Medical Imaging Analysis
• Natural Language Generation

Background:

• The integration of artificial intelligence (AI) in medical diagnostics is rapidly advancing.
• Multimodal large language models (LLMs) like GPT-4 show promise for automating clinical tasks.
• Evaluating the accuracy and potential biases of AI-generated medical reports is critical.

Purpose of the Study:

• To investigate the generative capabilities of the multimodal GPT-4 model.
• To compare AI-generated chest x-ray impressions with expert radiological assessments.
• To identify and analyze radiological bias in automated impression generation.

Main Methods:

• Utilized the multimodal GPT-4 model for generating chest x-ray impressions.
• Collected a dataset of chest x-rays and corresponding radiologist-written impressions.
• Employed automatic evaluation metrics and expert radiologist review for comparison.

Main Results:

• Significant discrepancies were observed between GPT-4 generated impressions and radiologist assessments.
• Automatic evaluation metrics did not fully capture the nuances of clinical accuracy.
• Evidence of radiological bias was detected in the AI-generated outputs.

Conclusions:

• Multimodal GPT-4 demonstrates generative capabilities but requires careful validation in clinical settings.
• Current automatic evaluation metrics may be insufficient for assessing AI-generated medical reports.
• Addressing radiological bias is essential for the responsible deployment of AI in medical imaging.