Active learning for clinical text classification: is it better than random sampling?

Area of Science:

• Medical informatics
• Machine learning
• Natural Language Processing

Background:

• Large labeled datasets are crucial for training effective medical text classification models.
• Active learning strategies aim to reduce the annotation burden by intelligently selecting informative data points for labeling.

Purpose of the Study:

• To evaluate the efficacy of active learning algorithms in reducing training set requirements for medical text classification.
• To compare the performance of distance-based (DIST), diversity-based (DIV), and combined (CMB) active learning algorithms against passive learning.
• To investigate the influence of dataset characteristics (diversity, uncertainty) on active learning algorithm performance.

Main Methods:

• Three active learning algorithms (DIST, DIV, CMB) were applied to five medical text datasets.
• Performance was assessed using classification accuracy and Area Under the ROC Curve (AUC) at varying sample sizes.
• Dataset diversity and uncertainty were quantified using relative entropy and correlated with algorithm performance.

Main Results:

• The DIST and CMB active learning algorithms outperformed passive learning across multiple datasets.
• DIST demonstrated superior performance over passive learning in all five datasets.
• Significant correlations were observed between dataset diversity and DIV performance, and dataset uncertainty and DIST performance.

Conclusions:

• Active learning algorithms can achieve performance comparable to passive learning with substantially smaller training sets in medical text classification.
• The DIV algorithm is more effective on diverse datasets, while the DIST algorithm performs better on datasets with lower uncertainty.