Related Experiment Video
Updated: Oct 21, 2025
03:14
Augmenting Large Language Models via Vector Embeddings to Improve Domain-Specific Responsiveness
Published on: December 6, 2024
764
Deep Learning-Based Natural Language Processing in Radiology: The Impact of Report Complexity, Disease Prevalence,
A W Olthof1,2,3, P M A van Ooijen4,5, L J Cornelissen4,6
1Department of Radiation Oncology, University of Groningen, University Medical Center Groningen, Hanzeplein 1, Groningen, The Netherlands. a.olthof@treant.nl.
Journal of Medical Systems
|September 4, 2021
Summary
Bidirectional Encoder Representations from Transformers (BERT) outperformed other deep learning models in analyzing radiology reports, showing stable performance across varying data sizes and prevalence. Understanding prevalence is key, as it affects sensitivity and specificity.
Area of Science:
- Medical Informatics
- Artificial Intelligence in Radiology
- Natural Language Processing
Background:
- Natural Language Processing (NLP) extracts valuable data from radiology reports for quality improvement and research.
- Factors like class imbalance, dataset size, report complexity, and algorithm choice impact NLP performance.
- A systematic, interrelated evaluation of these factors is lacking.
Purpose of the Study:
- To investigate the impact of dataset size and prevalence variations on the performance of four deep learning NLP models.
- To compare the performance of Dense, LSTM, CNN, and BERT algorithms in analyzing radiology reports.
Main Methods:
- Trained four deep learning NLP models (Dense, LSTM, CNN, BERT) on two datasets of radiology reports (trauma and chest imaging).
- Conducted 84 experiments for fracture data and 45 for chest data, varying training set size and prevalence.
- Evaluated model performance using sensitivity, specificity, positive predictive value, negative predictive value, AUC, and F score.
Main Results:
- All four deep learning architectures achieved high performance metrics (up to >0.90).
- BERT consistently outperformed Dense, LSTM, and CNN, demonstrating stable results across variations in training data size and prevalence.
- Prevalence significantly impacted sensitivity and specificity in opposing directions.
Conclusions:
- BERT demonstrates superior robustness and stability for NLP tasks in radiology compared to Dense, LSTM, and CNN.
- The prevalence of findings in radiology reports is a critical factor influencing NLP model performance, necessitating careful consideration in study design and interpretation.
