Benchmarking the most popular XAI used for explaining clinical predictive models: Untrustworthy but could be useful

Area of Science:

• Clinical Informatics
• Artificial Intelligence in Healthcare
• Medical Decision Support

Background:

• Clinical predictive models are increasingly used in healthcare.
• Explainable artificial intelligence (XAI) aims to make these models understandable to clinicians.
• Assessing the trustworthiness of XAI in clinical settings is crucial for adoption.

Purpose of the Study:

• To evaluate the practicality and trustworthiness of popular XAI methods for clinical predictive models.
• To determine if XAI explanations align with clinical reality and workflow.

Main Methods:

• Two common XAI methods were assessed using Electronic Medical Records (EMR) data from Australian hospitals.
• XAI explanations were evaluated for domain appropriateness, clinical workflow impact, and consistency.
• Explanations were benchmarked against actual clinical deterioration triggers, with agreement quantified.

Main Results:

• XAI methods demonstrated violations of consistency criteria.
• Moderate concordance (0.47-0.8) was found between XAI explanations and true clinical triggers.
• These findings indicate issues with reliability and actionability, impacting clinician trust.

Conclusions:

• Current XAI explanations are not sufficiently trustworthy to guide clinical interventions.
• XAI may assist in model troubleshooting and provide supplementary insights.
• Clinician involvement in XAI development and critical judgment are essential for safe implementation.