Large-Scale Evaluation of Machine Learning Models in Identifying Follow-Up Recommendations in Radiology Reports.

Area of Science:

• Radiology and Medical Informatics
• Artificial Intelligence in Healthcare
• Natural Language Processing

Background:

• Radiology reports contain critical follow-up recommendations for patient care and risk mitigation.
• Current methods for identifying these recommendations across diverse reports and modalities are limited.
• Open-source large language models (LLMs) offer potential for automated recommendation identification.

Purpose of the Study:

• To evaluate machine learning (ML) models, including LLAMA3 and GPT-4, for identifying follow-up recommendations in radiology reports.
• To compare the performance of various text classification methods on different sections of radiology reports.
• To assess the generalization capabilities of ML models on external and temporal datasets.

Main Methods:

• Retrospective analysis of 49,769 radiology reports from multiple imaging modalities and annotation methods.
• Evaluation of 32 text classification methods on 'findings' and 'impression' sections.
• Testing model generalization using the MIMIC-CXR database and institutional CT reports.

Main Results:

• A hybrid generative-discriminative model (Hybrid-google) achieved the highest F1 score (0.835) for the 'findings' section.
• An attention-based bidirectional LSTM (AttBiLSTM-random) achieved the highest F1 score (0.979) for the 'impression' section.
• GPT-4 with prefixed prompting showed strong generalization, with F1 scores of 0.969 (MIMIC-CXR) and 0.973 (institutional CT).

Conclusions:

• ML models demonstrate significant potential for automating the classification of follow-up recommendations in radiology reports.
• Different ML architectures excel in identifying recommendations from distinct report sections.
• Advanced models like GPT-4 offer robust performance for external and temporal data generalization.