Stability and robustness of minimal majority vote interpretable ensembles

Area of Science:

• Machine Learning
• Explainable AI (XAI)
• Ensemble Methods

Background:

• Minimal majority-vote ensembles are favored for their interpretability.
• However, the pursuit of minimality can lead to multiple solutions and instability.
• Decision stumps are simple decision trees often used in ensemble methods.

Purpose of the Study:

• To investigate the stability and robustness of minimal majority-vote ensembles of decision stumps.
• To define and apply novel metrics for evaluating ensemble stability and robustness.
• To assess the impact of sample size and feature noise on minimal ensemble performance.

Main Methods:

• Defined three metrics: multiplicity rate, bootstrap stability (Jaccard similarity), and feature-flip robustness.
• Evaluated metrics on synthetic datasets (binary [Formula: see text]-10, [Formula: see text]-500) and binarized UCI datasets.
• Utilized MILP-based solvers to confirm trends at larger sample sizes (n up to 500).
• Conducted revision analyses including label-noise sweeps and randomized tie-breaking.

Main Results:

• Minimal ensembles consistently fit data but show low bootstrap stability and high multiplicity in low-sample regimes.
• Robustness decreases gradually with increasing feature noise.
• On real-world datasets, minimal ensembles maintain accuracy but are sensitive to perturbations if stability is not prioritized.
• Larger sample sizes did not fully resolve stability issues.

Conclusions:

• Reporting stability alongside size is crucial for interpretable models.
• Minimality-only selection can result in brittle explanations, especially in critical applications.
• Explicitly considering stability is necessary for reliable minimal ensemble models.