SliceTeller: A Data Slice-Driven Approach for Machine Learning Model Validation.

Area of Science:

• Machine Learning Operations (MLOps)
• Data Science
• Software Engineering

Background:

• Real-world machine learning (ML) models require rigorous evaluation for product release, fairness across diverse groups, and consistent performance in varied scenarios.
• Identifying and mitigating issues within specific 'Data Slices' (e.g., weather conditions for autonomous driving, demographic groups for credit scoring) is crucial for reliable ML applications.
• Current MLOps cycles necessitate effective methods for discovering model failures, understanding their root causes, and implementing targeted improvements.

Purpose of the Study:

• To introduce SliceTeller, a novel tool designed for debugging, comparing, and enhancing machine learning models based on critical data slices.
• To provide mechanisms for automatically discovering problematic data slices and elucidating the reasons behind model failures.
• To present SliceBoosting, an efficient algorithm for optimizing model performance across prioritized data slices and to facilitate model version comparison.

Main Methods:

• Development of SliceTeller, a tool for automated data slice discovery and model debugging.
• Implementation of SliceBoosting algorithm for estimating optimization trade-offs across data slices.
• Evaluation of SliceTeller using three use cases, including two real-world product development scenarios.

Main Results:

• SliceTeller effectively discovers problematic data slices and aids in understanding model failure modes.
• The SliceBoosting algorithm provides efficient trade-off estimations for slice-specific optimization.
• The system empowers developers to compare model versions, leading to improved product-quality ML models.

Conclusions:

• SliceTeller offers a comprehensive solution for debugging, comparing, and improving machine learning models by focusing on critical data slices.
• The tool and its associated algorithm contribute to more robust, fair, and reliable ML model development within MLOps pipelines.
• Real-world application evaluations demonstrate SliceTeller's efficacy in enhancing the quality of ML models for product deployment.