Evaluating GPT-V4 (GPT-4 with Vision) on Detection of Radiologic Findings on Chest Radiographs.

Area of Science:

• Artificial intelligence in medical imaging
• Natural language processing for radiology
• Computer vision in healthcare

Background:

• Automated generation of radiologic findings from chest radiographs is crucial for medical image analysis.
• OpenAI's GPT-4 with vision (GPT-4V) presents potential for automated image-text pair generation.
• The efficacy of GPT-4V in real-world chest radiography requires thorough investigation.

Purpose of the Study:

• To evaluate the capability of GPT-4V in generating radiologic findings from real-world chest radiographs.
• To compare GPT-4V performance against a radiologist-annotated reference standard.
• To assess performance in both zero-shot and few-shot learning settings.

Main Methods:

• Retrospective study of 100 chest radiographs from NIH and MIDRC datasets.
• Radiologist annotation established a reference standard for clinical conditions and ICD-10 codes with laterality.
• GPT-4V performance assessed in zero-shot and few-shot settings, compared against the reference standard.

Main Results:

• In zero-shot setting, GPT-4V achieved low performance metrics (e.g., F1 scores of 7.3% on NIH, 18.2% on MIDRC) for ICD-10 code detection.
• Including laterality further reduced performance (e.g., F1 scores of 4.5% on NIH, 6.4% on MIDRC).
• Few-shot learning demonstrated improved true-positive rates and F1 scores but not substantial increases in positive predictive value.

Conclusions:

• GPT-4V, despite promise in natural image understanding, exhibited limited effectiveness in interpreting real-world chest radiographs.
• Current performance suggests GPT-4V is not yet suitable for automated generation of radiologic findings from chest X-rays.
• Further research and model development are needed to enhance AI capabilities in medical image analysis.