Evaluating active learning methods for annotating semantic predications.

Area of Science:

• Biomedical Informatics
• Machine Learning
• Natural Language Processing

Background:

• Filtering incorrect semantic predications from large biomedical databases like SemMedDB is crucial for data accuracy.
• Manual annotation is time-consuming and costly, necessitating efficient automated methods.

Purpose of the Study:

• To evaluate and compare various active learning (AL) strategies for filtering incorrect semantic predications in SemMedDB.
• To introduce and assess a novel combined AL strategy, dynamic beta, designed without hand-tuned hyperparameters.

Main Methods:

• Eight AL strategies (uncertainty, representative, combined) were tested on two SemMedDB datasets (substance interactions, clinical medicine).
• Performance was measured by Area Under the Learning Curve (ALC) and training examples needed for target performance.
• Query patterns of strategies were visualized and compared.

Main Results:

• All evaluated active learning methods outperformed the baseline.
• Combined strategies achieved the highest ALC, improving over baseline by >0.05 and reducing annotation effort by up to 58%.
• The proposed dynamic beta strategy demonstrated near-optimal performance across both datasets.

Conclusions:

• Active learning significantly reduces annotation costs for identifying erroneous semantic predications in SemMedDB.
• Strategies that efficiently sample representative data are more effective for this task.
• The novel dynamic beta active learning method shows potential for practical application.