Interpretability With Accurate Small Models.

Area of Science:

• Machine Learning
• Artificial Intelligence
• Data Science

Background:

• Interpretable models are crucial but often less accurate than larger ones.
• Constraining model size (e.g., decision tree depth) typically reduces classification accuracy.
• A trade-off exists between model interpretability and predictive performance.

Purpose of the Study:

• To present a practical technique for minimizing the trade-off between interpretability and classification accuracy.
• To enable arbitrary learning algorithms to produce highly accurate, small-sized models.
• To identify optimal training data distributions for a given model size.

Main Methods:

• Representing training distributions as combinations of sampling schemes, each defined by a parameterized probability mass function.
• Utilizing an Infinite Mixture Model with Beta components to represent the combination of sampling schemes.
• Employing Bayesian Optimization to learn mixture model parameters, reducing optimization variables to a fixed set of eight.

Main Results:

• Significantly reduced the number of variables needed for distribution optimization from O(d) to a fixed set of eight.
• Demonstrated effectiveness across decision trees, linear probability models, and gradient boosted models on real-world datasets.
• Achieved significant F1-score improvements, exceeding 100% in some cases.

Conclusions:

• The proposed model-agnostic technique effectively balances interpretability and accuracy for various model families.
• This method offers a flexible and efficient approach to generating high-performance, interpretable machine learning models.
• The technique requires relatively cheap preprocessing and yields substantial performance gains.