Extracting Pulmonary Embolism Diagnoses From Radiology Impressions Using GPT-4o: Large Language Model Evaluation...

Area of Science:

• Natural Language Processing
• Artificial Intelligence in Medicine
• Radiology Informatics

Background:

• Pulmonary embolism (PE) is a life-threatening condition requiring prompt diagnosis.
• Manual extraction of PE diagnoses from radiology reports is labor-intensive.
• Automated solutions using advanced NLP models are needed to enhance efficiency and accuracy.

Purpose of the Study:

• To develop and evaluate an automated system using GPT-4o for extracting PE diagnoses from radiology report impressions.
• To enhance clinical decision-making and streamline healthcare workflows.
• To compare GPT-4o's performance against a baseline Clinical Longformer model.

Main Methods:

• Two approaches were developed: a fine-tuned Clinical Longformer and a GPT-4o-based extractor.
• Clinical Longformer, an encoder-only model, was trained on 1000 radiology reports.
• GPT-4o, a decoder-only LLM, was validated on 200 reports, with postdeployment assessment on 200 operational records.

Main Results:

• GPT-4o demonstrated superior performance over Clinical Longformer, achieving a sensitivity of 1.0 and an F1-score of 0.975 on the validation set.
• Postdeployment evaluation showed GPT-4o maintained high accuracy with 1.0 sensitivity, 0.94 specificity, and 0.97 F1-score.
• The high accuracy suggests a significant reduction in manual review and improved diagnostic precision.

Conclusions:

• GPT-4o offers an effective and reliable method for automated PE diagnosis extraction from radiology reports.
• This technology can significantly improve patient outcomes by expediting diagnosis and treatment.
• The system has the potential to revolutionize clinical decision-making for critical conditions.