An objective framework for evaluating unrecognized bias in medical AI models predicting COVID-19 outcomes

Area of Science:

• Medical Artificial Intelligence
• Clinical Machine Learning
• Health Informatics

Background:

• The integration of AI/ML into clinical settings poses risks of harm due to modeling bias.
• Bias in medical AI applications is often incompletely measured, necessitating robust evaluation frameworks.

Purpose of the Study:

• To provide a framework for objectively evaluating medical AI, specifically binary classification models.
• To assess unrecognized bias in AI models used for predicting severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) outcomes.

Main Methods:

• Evaluation of 4 AI models using data from over 56,000 Mass General Brigham patients.
• Models predicted risks of hospital admission, ICU admission, mechanical ventilation, and death post-SARS-CoV-2 infection.
• Retrospective and prospective evaluation using model-level metrics and a novel individual-level error metric.

Main Results:

• Inconsistent instances of model-level bias were observed across the prediction models.
• A trend of slightly higher error rates was identified for older patients at the individual level.
• AI models exhibited varied performance across different protected subpopulations.

Conclusions:

• Holistic evaluation is essential to uncover unrecognized AI bias.
• Comprehensive bias assessment is needed to inform investigations into the root causes of bias.
• Objective AI evaluation can drive improvements in fairness and equity in medical AI applications.