Detecting Laterality Errors in Combined Radiographic Studies by Enhancing the Traditional Approach With GPT-4o:...

Area of Science:

• Medical Informatics
• Natural Language Processing in Healthcare
• Radiology Quality Assurance

Background:

• Laterality errors in radiology reports pose significant patient safety risks.
• Screening methods for these errors in combined radiographic reports are underdeveloped.

Purpose of the Study:

• To analyze the challenges of the combined radiographic report format.
• To introduce and evaluate a novel ensemble method (rule-based + GPT-4o) for laterality error detection.
• To assess performance differences between real-world imbalanced and synthetic balanced datasets.

Main Methods:

• Retrospective analysis of 10,000 deidentified radiology reports.
• Development and comparison of baseline, workaround, and GPT-4o-augmented rule-based methods.
• Evaluation of fine-tuned RoBERTa, ClinicalBERT, and GPT-4o models on real-world and synthetic datasets.

Main Results:

• Laterality error rate was 1.20% in real-world reports, higher in combined (1.47%) vs. non-combined (0.57%) reports.
• The rule-based+GPT-4o method achieved the highest recall on imbalanced real-world data, outperforming GPT-4o, ClinicalBERT, and RoBERTa.
• Significant performance drops in precision and F1-scores were observed for all models on real-world imbalanced data compared to synthetic balanced data.

Conclusions:

• The combined radiographic report format presents unique challenges for quality assurance and NLP.
• The rule-based+GPT-4o ensemble method demonstrates effectiveness in detecting laterality errors in real-world, imbalanced datasets.
• Future research must incorporate real-world imbalanced data to accurately benchmark performance.