Auditing fairness in clinical AI systems using provenance-based simulation: a comparative and regulatory perspective

Area of Science:

• Medical Informatics
• Artificial Intelligence
• Health Equity

Background:

• Algorithmic bias and lack of transparency hinder Artificial Intelligence (AI) adoption in clinical decision support.
• Data provenance is crucial for auditing AI fairness and ensuring trustworthy AI systems.

Purpose of the Study:

• To develop and evaluate an AI auditing framework using detailed data provenance to assess fairness in clinical decision support models.
• To compare the fairness of logistic regression and random forest models using a synthetic patient dataset.

Main Methods:

• Simulated audits on a synthetic patient dataset (N=1,000) using logistic regression and random forest models.
• Employed 5-fold cross-validation and permutation testing to detect gender bias.
• Utilized detailed data provenance logs to trace and quantify bias.

Main Results:

• Provenance logs successfully detected gender biases in both logistic regression and random forest models.
• Logistic regression showed statistically significant bias (EOD = +0.256, p = 0.0080), while random forest's bias was not statistically significant (EOD = +0.055, p = 0.5664).
• Random forest demonstrated 57% less bias than logistic regression, despite lower overall accuracy.

Conclusions:

• The developed provenance-based auditing framework effectively distinguishes systematic discrimination from random variation in AI models.
• A standardized AI Fairness Provenance Record supports regulatory compliance (FDA, ONC) and promotes responsible AI in clinical settings.
• Fairness in AI is not solely dependent on model interpretability; auditing with data provenance is essential.