Transformer versus traditional natural language processing: how much data is enough for automated radiology report...

Area of Science:

• Medical Informatics
• Natural Language Processing
• Machine Learning

Background:

• State-of-the-art natural language processing (NLP) utilizes transformer deep-learning architectures, requiring extensive training data.
• The efficacy of these models on smaller, specialized datasets, such as radiology reports, remains a critical question.

Purpose of the Study:

• To compare the performance of transformer-based NLP models against traditional machine learning models for various radiology report classification tasks.
• To determine optimal model selection based on training dataset size and characteristics.

Main Methods:

• Evaluated BioBERT (transformer) against gradient boosted trees, random forests, and logistic regression on seven radiology report classification tasks.
• Trained and tested models on the full dataset (7,204 reports) and random subsets (2.5% to 75%) to assess performance variations with data size.

Main Results:

• BioBERT underperformed traditional models on smaller training datasets (<1,000 reports).
• Transformer models required approximately 1,000 reports to match or exceed traditional model performance.
• Model performance plateaued around 1,250-1,500 training samples, indicating diminishing returns with increased data.

Conclusions:

• Traditional NLP techniques demonstrate superior performance for smaller (<1,000 reports) and imbalanced radiology report datasets.
• Transformer models achieve better performance with larger datasets but are less efficient with limited data.
• Findings provide benchmarks for selecting appropriate NLP models based on dataset size in clinical research.