Inter-labeler and intra-labeler variability of condition severity classification models using active and passive...

Area of Science:

• Machine Learning
• Medical Informatics
• Computational Biology

Background:

• Expert labeling for clinical classification is time-consuming and costly.
• Active learning (AL) methods can reduce labeling efforts.
• Variability in labeler expertise poses challenges for learning methods.

Purpose of the Study:

• To evaluate the effect of AL methods on intra-labeler and inter-labeler variability.
• To examine learning from a consensus label provided by multiple labelers.

Main Methods:

• Utilized the CAESAR-ALE framework for clinical condition severity classification.
• Compared three AL methods (SVM-Margin, Exploitation, Combination_XA) against passive learning.
• Analyzed classification performance variance using labels from seven experts on a dataset of 516 conditions from 1.9 million patients.

Main Results:

• AL methods produced smoother intra-labeler learning curves and significantly lower intra-labeler variability (p=0.049).
• AL methods reduced inter-labeler AUC standard deviation by nearly half compared to passive learning.
• Using a consensus label with AL methods reduced intra-labeler AUC variance during learning.

Conclusions:

• AL methods decrease intra-labeler variability, reducing the risk of suboptimal model performance.
• AL methods reduce inter-labeler performance variance, lessening reliance on specific labelers.
• Consensus labels with AL methods offer performance comparable to gold standards and superior to random selection.