Applying active learning to assertion classification of concepts in clinical text.

Area of Science:

• Clinical Natural Language Processing (NLP)
• Machine Learning
• Medical Informatics

Background:

• Supervised machine learning for clinical NLP requires extensive annotated data, which is costly due to physician involvement.
• Active learning offers a solution by strategically selecting samples to reduce annotation burden while maintaining model quality.

Purpose of the Study:

• To apply and evaluate active learning strategies for a clinical text classification task: determining assertion status of clinical concepts.
• To compare the effectiveness of active learning against passive learning (random sampling) in this clinical NLP context.

Main Methods:

• Utilized the annotated corpus from the 2010 i2b2/VA Clinical NLP Challenge for assertion classification.
• Implemented and assessed various existing and novel active learning algorithms.
• Evaluated performance using the global ALC score, derived from the Area Under the average Learning Curve of the AUC score.

Main Results:

• Active learning strategies outperformed passive learning, achieving a higher best ALC score (0.7715) compared to random sampling (ALC-0.7411).
• Active learning required substantially fewer samples to reach target classification performance.
• Achieved a 62.5% reduction in manual annotation effort (12 vs. 32 samples) to reach an AUC of 0.79.

Conclusions:

• Active learning is an effective approach for clinical NLP tasks, significantly reducing annotation costs.
• The findings demonstrate the potential of active learning to improve efficiency in building high-quality clinical NLP models.