The CrowdGleason dataset: Learning the Gleason grade from crowds and experts

Area of Science:

• Digital pathology
• Machine learning in cancer diagnostics
• Computational pathology

Background:

• Prostate cancer (PCa) diagnosis relies on manual analysis of Whole Slide Images (WSIs) for Gleason scoring, a process that is subjective and time-consuming.
• Automated systems using machine learning show promise for assisting pathologists, but require large, labeled datasets for training.
• Scarcity of expert-annotated data is a major bottleneck for developing and validating these AI systems.

Purpose of the Study:

• To introduce the CrowdGleason dataset, a novel resource for prostate histopathology.
• To evaluate crowdsourcing-based Gaussian Process (GP) methods for Gleason grade prediction using this dataset.
• To compare the performance of crowdsourcing methods against expert-labeled data and individual pathologist performance.

Main Methods:

• Development of the CrowdGleason dataset comprising 19,077 patches from 1045 WSIs, annotated via a crowdsourcing protocol with seven pathologists-in-training.
• Evaluation of two GP-based crowdsourcing methods: SVGPCR (trained on CrowdGleason) and SVGPMIX (combining CrowdGleason and SICAPv2 datasets).
• Comparative analysis against other crowdsourcing aggregation techniques and expert-labeled datasets.

Main Results:

• The GP-based crowdsourcing approach (SVGPCR) outperformed other aggregation methods (κ=0.7048±0.0207).
• SVGPCR trained with crowdsourced labels surpassed GP trained with expert labels from SICAPv2 (κ=0.6583±0.0220) and most individual pathologists (mean κ=0.5432).
• SVGPMIX, combining both datasets, achieved the highest performance (κ=0.7814±0.0083 on SICAPv2 and κ=0.7276±0.0260 on CrowdGleason).

Conclusions:

• The CrowdGleason dataset is suitable for training and validating supervised and crowdsourcing AI methods for prostate cancer diagnosis.
• Crowdsourcing methods trained on this dataset achieve competitive results compared to expert-labeled data.
• Combining expert and non-expert annotations offers a promising avenue for large-scale data labeling in computational pathology.