Explainable AI needs formalization

Area of Science:

• Computer Science
• Artificial Intelligence
• Machine Learning

Background:

• Explainable artificial intelligence (XAI) aims to make machine learning (ML) decisions human-understandable.
• Current XAI methods face scrutiny due to limitations in reliably answering questions about ML models, data, or inputs.

Purpose of the Study:

• To critically evaluate the current state of XAI methods.
• To identify fundamental reasons for the limitations of existing XAI techniques.
• To propose a path forward for developing more reliable and validated XAI algorithms.

Main Methods:

• Analysis of common XAI method shortcomings.
• Identification of systematic errors in feature importance attribution.
• Conceptual framework for defining explanation correctness.

Main Results:

• Popular XAI methods incorrectly attribute importance to independent input features.
• This inaccuracy limits XAI's utility for model diagnosis, scientific discovery, and intervention targeting.
• Current XAI methods lack well-defined problems and objective evaluation criteria.

Conclusions:

• XAI methods need formal problem definition and use-case-dependent evaluation criteria.
• Developing objective metrics for explanation performance is crucial for validating XAI algorithms.
• Future XAI research should focus on addressing well-defined problems with rigorous validation.