Boosting wisdom of the crowd for medical image annotation using training performance and task features

Area of Science:

• Medical Artificial Intelligence
• Machine Learning
• Dermatology

Background:

• High-quality labeled medical datasets are a significant bottleneck in developing medical AI.
• Crowdsourcing platforms can recruit diverse individuals for data labeling tasks.

Purpose of the Study:

• To evaluate various wisdom of the crowd algorithms for labeling medical images.
• To determine the optimal approach for leveraging crowd intelligence in medical AI.

Main Methods:

• Recruited individuals via an app to classify skin lesions from the International Skin Lesion Challenge 2018 into 7 categories.
• Tested multiple wisdom of the crowd algorithms, ranging from simple averages to complex Bayesian models.
• Employed switchboard analysis to assess algorithm performance based on individual characteristics and task environment.

Main Results:

• The best-performing algorithms selected top performers, weighted decisions by training accuracy, and considered the task environment.
• These sophisticated crowd-based labeling methods significantly surpassed expert performance.
• Individual variability in experience and performance was a key factor influencing algorithm effectiveness.

Conclusions:

• Wisdom of the crowd algorithms, particularly those incorporating performance weighting and error analysis, can effectively generate high-quality labeled medical data.
• These findings have significant implications for accelerating the development of robust medical AI systems.
• Optimizing crowd-based labeling strategies is crucial for overcoming data limitations in medical AI.