A Pathologist-Annotated Dataset for Validating Artificial Intelligence: A Project Description and Pilot Study.

Area of Science:

• Digital pathology
• Artificial intelligence in oncology
• Computational pathology

Background:

• Validating AI algorithms for clinical use in medical imaging is hindered by a lack of standardized ground truth data.
• Current research often prioritizes novel algorithm development over dataset creation for validation.
• Accurate quantification of stromal tumor-infiltrating lymphocytes (sTILs) is crucial for breast cancer prognosis and treatment response.

Purpose of the Study:

• To establish a robust validation dataset of pathologist annotations for AI algorithms processing whole slide images.
• To focus on efficient data collection and evaluation methodologies for AI algorithm performance.
• To specifically address the estimation of stromal tumor-infiltrating lymphocytes (sTILs) density in breast cancer.

Main Methods:

• Digitized 64 whole slide images of invasive ductal carcinoma core biopsies.
• Selected 10 regions of interest (ROIs) per slide for annotation by pathologists.
• Developed training materials and workflows for crowdsourced annotations via optical microscopy and digital platforms, collecting ROI appropriateness and sTIL density.

Main Results:

• 19 pathologists contributed 1645 ROI evaluations.
• The pilot study generated sufficient data for nominal sTIL infiltration cases.
• Identified correlations in sTIL densities within cases and significant pathologist variability, informing future sampling and statistical methods.

Conclusions:

• Successfully developed and piloted workflows for efficient data collection for AI algorithm validation.
• The dataset will be utilized as an external validation tool for algorithms.
• Plans include seeking regulatory feedback and sharing the dataset, methods, and lessons learned for broader community benefit.