Cost-aware active learning for named entity recognition in clinical text.

Area of Science:

• Machine Learning
• Natural Language Processing
• Bioinformatics

Background:

• Active Learning (AL) aims to minimize annotation costs by selecting informative samples.
• Existing AL methods often overlook the varying costs associated with annotating different samples.
• This limitation is particularly relevant in specialized domains like clinical text analysis.

Purpose of the Study:

• To develop and evaluate a novel cost-aware Active Learning (AL) algorithm.
• To integrate annotation cost estimation into the AL sample selection process.
• To assess the performance of the cost-aware AL method in both simulated and real-world user studies.

Main Methods:

• A cost-aware AL algorithm, Cost-CAUSE, was designed, incorporating lexical and syntactic features to estimate annotation costs.
• The Cost-CAUSE algorithm was integrated into an existing AL framework.
• Performance was evaluated using the 2010 i2b2/VA dataset through simulation studies and a user annotation task, comparing against non-cost-aware AL and passive learning.

Main Results:

• The developed cost model demonstrated a good fit with empirical annotation data.
• Cost-CAUSE improved simulation area under the learning curve (ALC) scores by up to 5.6% compared to random sampling and 4.9% compared to other AL methods.
• In user studies, Cost-CAUSE achieved higher ALC scores and reduced annotation time by 20.5%-30.2% compared to passive learning.

Conclusions:

• Cost-CAUSE effectively reduces annotation costs compared to random sampling.
• Real-world AL performance is influenced by factors beyond sample informativeness, including user accuracy and condition.
• The cost-aware approach offers a practical improvement for efficient data annotation in clinical NLP tasks.