Practical Evaluation of ChatGPT Performance for Radiology Report Generation.

Area of Science:

• Medical Informatics
• Artificial Intelligence in Healthcare
• Natural Language Processing

Background:

• Radiology report generation is inefficient, prone to errors and inconsistencies.
• Natural Language Processing (NLP) offers potential solutions to streamline this process.

Purpose of the Study:

• To evaluate the efficacy of ChatGPT, a large language model (LLM), in generating radiology reports.
• To compare the performance of various NLP models in mimicking physician-authored reports.

Main Methods:

• Utilized 1000 chest X-ray reports from the MIMIC database.
• Employed Claude.ai for initial keyword extraction and ChatGPT for report generation using a 3-step prompt.
• Assessed report similarity using lexical and sentence similarity techniques against physician reports.

Main Results:

• Bidirectional and Auto-Regressive Transformers (Bart) and Cross-lingual Language Model (XLM) achieved high similarity (up to 99.3%) with physician reports.
• Decoding-enhanced BERT with disentangled attention (DeBERTa) and sequence-matching models showed lower alignment.
• The Word-Embedding model demonstrated strong performance (84.4% similarity) in the Impression section.

Conclusions:

• Significant variations exist in NLP model performance for radiology report generation.
• Careful selection and evaluation of NLP models are essential for clinical application.
• ChatGPT demonstrates potential to enhance efficiency and accuracy in radiology reporting.