Crowdsourcing image annotation for nucleus detection and segmentation in computational pathology: evaluating experts,...

Area of Science:

• Computational pathology
• Digital pathology
• Biomedical image analysis

Background:

• High-quality annotated images are crucial for developing computational pathology tools.
• Expert-derived annotations are time-consuming and costly.
• Crowdsourcing offers a potential solution for rapid image annotation.

Purpose of the Study:

• To explore crowdsourcing for nucleus detection and segmentation in computational pathology.
• To compare the concordance of crowdsourced annotations with expert and automated methods.
• To assess the impact of annotator skill and image size on crowdsourced annotation performance.

Main Methods:

• Crowdsourcing experiments were conducted on the CrowdFlower platform for nucleus detection and segmentation.
• Annotations were obtained from crowdsourced contributors (levels 1-3), research fellows, and expert pathologists.
• Automated methods were also used for comparison.
• Concordance was evaluated against expert pathologist-derived annotations.

Main Results:

• Crowdsourced annotations were generated significantly faster and at a lower cost than traditional methods.
• For nucleus detection, research fellow annotations showed the highest concordance with experts, followed closely by crowdsourced annotations.
• For nucleus segmentation, crowdsourced annotations (especially aggregated consensus) and research fellow annotations outperformed automated methods.
• Crowdsourced performance decreased with larger image sizes.

Conclusions:

• Crowdsourcing to non-experts is a viable method for large-scale annotation in computational pathology.
• It enables rapid generation of labeled images for algorithm development and evaluation.
• Optimizing crowdsourcing strategies, such as aggregation and considering image size, is key for effective application.