Task-Based Sampling of Patient Data for Rigorous Machine Learning/AI Performance Assessment

Area of Science:

• Medical informatics
• Artificial intelligence in healthcare
• Data science

Background:

• AI algorithm performance assessment requires independent datasets representative of the intended clinical population.
• Using all available data can be impractical and may introduce sampling bias.
• Representative data is crucial for reliable AI model training and validation.

Purpose of the Study:

• To develop and demonstrate a computational method for task-based data sampling from large repositories.
• To generate datasets matched to specific demographic and clinical profiles for AI performance assessment.
• To mitigate sampling bias in AI algorithm development and evaluation.

Main Methods:

• A task-based sampling algorithm was developed, requiring users to define an initial cohort, target distribution, and allowable deviation.
• The algorithm was applied to the Medical Imaging and Data Resource Center (MIDRC) data commons.
• Demographic characteristics and disease states were used as clinical attributes for matching to an intended population profile (e.g., CDC demographics).

Main Results:

• The algorithm successfully sampled cohorts (542 and 870 patients) from an initial >4000 patient cohort.
• Sampled cohorts closely matched the target demographic distribution with low average clinical attribute differences (1.0% and 2.1%).
• The method demonstrated effectiveness in generating matched samples for AI performance assessment.

Conclusions:

• The developed task-based sampling algorithm effectively generates matched samples from large datasets.
• This approach reduces sampling bias, enhancing the reliability of AI algorithm training and performance assessment.
• The method provides a valuable tool for creating representative datasets in medical AI research.