Impossibility theorems for feature attribution.

Area of Science:

• Artificial Intelligence
• Machine Learning
• Explainable AI (XAI)

Background:

• Numerous interpretability methods exist, but empirical failures raise concerns about their practical application.
• Practitioners lack clear guidance on principled usage and selection of feature attribution methods.

Purpose of the Study:

• To demonstrate theoretical limitations of complete and linear feature attribution methods for moderately rich model classes.
• To highlight the importance of end-task definition in evaluating interpretability methods.

Main Methods:

• Theoretical analysis of feature attribution methods under specific assumptions (completeness, linearity).
• Investigation across common end-tasks: local behavior characterization, spurious feature identification, and algorithmic recourse.
• Comparison with a baseline approach of repeated model evaluations.

Main Results:

• Complete and linear feature attribution methods can perform no better than random guessing for inferring model behavior.
• These limitations apply to widely used methods such as Integrated Gradients and Shapley Additive Explanations (SHAP).
• A simple strategy of repeated model evaluations can outperform complex methods when the end-task is well-defined.

Conclusions:

• Current feature attribution methods may not reliably improve understanding of complex models.
• The precise definition of the end-task is crucial for effective model interpretation.
• Well-defined end-tasks enable simpler, direct evaluation strategies that can be more effective than sophisticated feature attribution techniques.