HISTAI: a valuable dataset with a valuable lesson.

Area of Science:

• Computational pathology
• Artificial intelligence in medicine
• Digital pathology

Background:

• Artificial intelligence (AI) in pathology requires high-quality, validated whole slide image (WSI) datasets.
• Existing WSI datasets are often scarce, lack diversity, and are not well-validated, limiting AI progress.
• The HISTAI resource offers a large, open-source collection of WSIs with clinical metadata.

Purpose of the Study:

• To conduct a pathologist-led evaluation of the HISTAI dataset's label accuracy, metadata completeness, and composition.
• To identify limitations and potential challenges for using the HISTAI resource in AI development.
• To assess the clinical reliability of the HISTAI dataset for computational pathology applications.

Main Methods:

• Pathologist review of 328 selected cases from the HISTAI resource.
• Analysis of label accuracy, metadata completeness (demographics, specialty), and dataset composition.
• Focused review of diagnostic conclusion concordance, molecular annotation, and adherence to WHO CNS5 criteria for specific tumor types.

Main Results:

• Identified fewer unique cases than reported, with incomplete demographic data (55%).
• Uneven dataset composition (dermatopathology 47.1%, gastrointestinal 24.0%) with poorly reported specialties.
• Significant discrepancies between diagnosis and conclusion fields (20.7% concordance, 27.1% conflict), ambiguous conclusions (30.3%), incomplete molecular data (18.9%), and non-compliance with WHO CNS5 criteria for gliomas.

Conclusions:

• The HISTAI dataset exhibits substantial ambiguities in ground-truth labeling and incomplete molecular annotation.
• Limited documentation of dataset provenance and ethical oversight requires attention.
• Effective and responsible use of HISTAI necessitates rigorous clinical validation and pathologist-AI researcher collaboration.