Gamified Crowdsourcing as a Novel Approach to Lung Ultrasound Data Set Labeling: Prospective Analysis

Area of Science:

• Medical Imaging
• Machine Learning
• Crowdsourcing

Background:

• Machine learning (ML) models require high-quality labeled data for accurate medical diagnoses.
• Crowdsourced labeling offers a solution but faces challenges in ensuring label quality.

Purpose of the Study:

• To evaluate if a gamified crowdsourcing platform can produce expert-quality labels for medical imaging data.
• To assess the impact of continuous performance assessment, user feedback, and incentives on label quality.

Main Methods:

• 2384 lung ultrasound clips from 203 patients were used.
• Experts created reference standards; crowdsourced opinions were collected via the DiagnosUs app.
• Crowd labels were filtered, aggregated, and compared to expert labels for concordance.

Main Results:

• Crowdsourced label concordance (87.9%) was comparable to individual expert concordance (85.0%) on a test set.
• Crowd concordance (87.4%) exceeded individual expert concordance (80.8%) when compared against majority-voted reference standards.
• A small subset of quality-filtered crowd opinions (7) was sufficient for high concordance.

Conclusions:

• Gamified crowdsourcing can achieve expert-level accuracy for B-line classification on lung ultrasound images.
• This method provides an efficient way to generate labeled datasets for training ML systems in healthcare.