An active learning-enabled annotation system for clinical named entity recognition.

Area of Science:

• Natural Language Processing (NLP)
• Machine Learning
• Medical Informatics

Background:

• Active learning (AL) minimizes annotation costs for NLP models.
• Few studies explore AL in real-world medical settings.
• Clinical Named Entity Recognition (NER) is crucial for medical data.

Purpose of the Study:

• Develop the first AL-enabled annotation system for clinical NER.
• Evaluate a novel AL algorithm in simulation and real-world user studies.
• Assess AL performance in clinical NER model development.

Main Methods:

• Developed a novel AL algorithm for clinical NER.
• Conducted simulation studies to evaluate the AL algorithm.
• Performed user studies with nurses using the AL annotation system.

Main Results:

• Simulation showed the novel AL algorithm outperformed traditional AL and random sampling.
• User studies indicated AL methods were not consistently better than random sampling.
• Annotation time for actively selected sentences offset information gain.

Conclusions:

• Annotation time is a critical factor not addressed by current AL querying algorithms.
• Future work will focus on AL algorithms incorporating annotation time estimation.
• Further evaluation with more users is needed to validate AL system performance.