Crowdsourcing airway annotations in chest computed tomography images

Area of Science:

• Medical Imaging
• Computational Biology
• Crowdsourcing

Background:

• Manual airway measurement in chest CT scans is crucial for disease characterization (e.g., cystic fibrosis) but is labor-intensive.
• Machine learning for medical image analysis requires large, annotated datasets, which are often difficult to obtain.
• Crowdsourcing presents a potential solution for large-scale data annotation in medical imaging.

Purpose of the Study:

• To investigate the feasibility and accuracy of using crowdsourcing for annotating airways in chest computed tomography (CT) scans.
• To compare crowdsourced airway measurements with expert annotations.
• To assess the potential of crowdsourcing for generating data for machine learning models in respiratory disease research.

Main Methods:

• Image slices from 24 subjects with known airway locations were generated from chest CT scans.
• Crowd workers were tasked with outlining airway lumens and walls.
• Annotations were combined, and measurements were compared against expert-defined measurements, with exclusions for potentially erroneous annotations.

Main Results:

• Moderate to strong correlations were observed between crowdsourced and expert airway measurements after excluding unreliable annotations.
• Crowdsourced correlations were slightly lower than inter-expert correlations.
• Significant variability in results across subjects indicated challenges in achieving consistent annotation quality.

Conclusions:

• Crowdsourcing demonstrates potential for annotating airways in CT scans, offering a scalable alternative to manual methods.
• Further refinement of instructions and quality control mechanisms is necessary to improve robustness and reliability for practical applications.
• The study provides reproducible data and code for future research in crowdsourced medical image annotation.