"Who experiences large model decay and why?" A Hierarchical Framework for Diagnosing Heterogeneous Performance Drift

Area of Science:

• Machine Learning
• Artificial Intelligence
• Data Science

Background:

• Machine learning (ML) models often degrade in performance when deployed in new environments.
• Performance degradation is typically non-uniform, impacting certain subgroups more severely than others.
• Existing methods lack detailed insights into the causes and locations of performance decay across subgroups.

Purpose of the Study:

• To introduce a novel framework for identifying and understanding performance decay in ML models across different subgroups.
• To enable the design of targeted corrective actions by pinpointing specific subgroups and the reasons for their performance decline.
• To address the limitations of current approaches that focus on average performance shifts or simply identify affected subgroups without explanation.

Main Methods:

• Development of the Subgroup-scanning Hierarchical Inference Framework for performance drifT (SHIFT).
• SHIFT employs a two-stage approach: first identifying subgroups with significant performance decay, then investigating the underlying causes (covariate/outcome shifts).
• Evaluation of SHIFT's ability to identify interpretable subgroups and suggest targeted mitigation strategies.

Main Results:

• SHIFT successfully identifies specific subgroups experiencing disproportionately large performance decay.
• The framework provides interpretable explanations for performance degradation by analyzing variable-specific shifts.
• Real-world experiments demonstrate that SHIFT-guided actions effectively mitigate performance decay.

Conclusions:

• SHIFT offers a powerful tool for diagnosing and addressing performance disparities in deployed ML models.
• Understanding subgroup-specific performance decay is crucial for developing effective and targeted ML model maintenance strategies.
• The framework facilitates the creation of more robust and equitable AI systems by enabling precise interventions.