Does Cohort Selection Affect Machine Learning from Clinical Data?

Area of Science:

• Clinical Informatics
• Machine Learning in Healthcare
• Health Equity Research

Background:

• Machine learning (ML) models are increasingly used for predicting patient outcomes.
• Clinical data preprocessing involves critical decisions that can influence model performance.
• The National COVID Cohort Collaborative (N3C) provides a large dataset for studying these effects.

Purpose of the Study:

• To investigate the impact of cohort selection strategies on ML model quality and fairness.
• To analyze how arbitrary data processing decisions affect model predictions.
• To assess biases related to social determinants of health in ML models.

Main Methods:

• Experiments conducted using the N3C dataset.
• Generation of 16 distinct datasets by making four arbitrary cohort selection decisions.
• Evaluation of dataset variations in size and properties.
• Assessment of ML model performance across different cohorts.

Main Results:

• Significant differences observed in dataset characteristics based on inclusion/exclusion criteria.
• High potential for bias introduced by arbitrary cohort selection.
• Substantial variations in ML model performance when trained on different cohorts.
• Disparities in model performance highlighted when comparing cohorts with differing inclusion criteria.

Conclusions:

• Cohort selection is a critical factor influencing ML model bias and fairness.
• Transparent and justified data processing decisions are essential for reliable clinical ML.
• Further research is needed to mitigate biases associated with social determinants of health in ML models.