Comparison of active learning algorithms in classifying head computed tomography reports using bidirectional encoder...

Area of Science:

• Medical Informatics
• Radiology
• Artificial Intelligence

Background:

• Natural language processing (NLP) systems can aid radiologists by reducing missed findings in medical reports.
• High annotation costs pose a significant challenge in developing these NLP systems.
• Head computed tomography (CT) report analysis requires accurate significance estimation.

Purpose of the Study:

• To compare the effectiveness of active learning (AL) algorithms in NLP for estimating the significance of head CT reports.
• To evaluate different AL strategies using bidirectional encoder representations from transformers (BERT).

Main Methods:

• Utilized 3728 head CT reports categorized by importance.
• Employed UTH-BERT, a pre-trained BERT model.
• Compared seven sampling metrics: random sampling (RS), four uncertainty sampling (US) methods (least confidence, margin sampling, ratio of confidence, entropy sampling), and two distance-based sampling (DS) methods (cosine and Euclidian distance).
• Initiated with an empty labeled dataset and iteratively added 25 reports from an unlabeled data pool.

Main Results:

• Uncertainty sampling (US) methods showed significantly higher accuracy than random sampling (RS) when the labeled dataset contained fewer than 1800 reports.
• Distance-based sampling (DS) methods performed significantly lower than RS.
• Margin sampling (MS) and ratio of confidence (RC) were the most effective US methods.
• US methods reduced the required labeled data by 15.4-40.5%, with RC being the most efficient.
• US methods tended to prioritize minor categories for labeling earlier than RS and DS.

Conclusions:

• Uncertainty sampling (US) methods, specifically RC and MS, enable effective fine-tuning of BERT models for head CT report classification.
• These AL strategies help mitigate category imbalance in datasets.
• Active learning offers a valuable approach for efficient data annotation in larger-scale studies.