Evaluation of Artificial Intelligence-Based Gleason Grading Algorithms "in the Wild".

Area of Science:

• Pathology
• Artificial Intelligence
• Oncology

Background:

• The biopsy Gleason score is crucial for prostate cancer prognosis but suffers from pathologist variability.
• Artificial intelligence (AI) shows promise in matching and potentially improving pathologist accuracy for Gleason grading.
• Current AI algorithm evaluations are often limited to benchmark datasets, hindering comparison between research and commercial tools.

Purpose of the Study:

• To evaluate and compare the performance of leading public and commercial AI algorithms for Gleason grading.
• To assess AI performance against pathologist agreement using a diverse, real-world dataset of prostate biopsy images.

Main Methods:

• A crowdsourced dataset of whole-slide prostate biopsy images with varied Gleason scores was curated.
• Predictions from 5 top public AI algorithms (PANDA challenge) and 2 commercial AI algorithms were obtained.
• 10 pathologists evaluated the dataset in a reader study to establish a performance benchmark.

Main Results:

• Both public and commercial AI algorithms demonstrated high agreement with pathologist Gleason scores (quadratic kappa 0.617–0.900).
• Pathologist agreement ranged from 0.777 to 0.916 (pairwise quadratic weighted kappa).
• Commercial AI algorithms performed on par with or exceeded the performance of top-ranked public AI algorithms.

Conclusions:

• AI algorithms, both public and commercial, can achieve high accuracy in prostate cancer Gleason grading.
• Commercial AI tools demonstrate robust performance, comparable or superior to leading research algorithms in real-world clinical settings.
• AI has the potential to standardize and improve the accuracy of prostate cancer grading, aiding in patient prognostication.