A greedy stacking algorithm for model ensembling and domain weighting.

Area of Science:

• Machine Learning
• Biomedical Informatics
• Statistical Modeling

Background:

• Ensemble methods like stacking improve predictive accuracy by combining multiple models.
• Traditional linear stacking methods can struggle with highly correlated predictions.
• A need exists for efficient, interpretable stacking algorithms in complex data environments.

Purpose of the Study:

• To develop and evaluate a novel greedy algorithm for model stacking.
• To address limitations of linear stacking, particularly with correlated predictions.
• To assess the algorithm's performance across diverse biomedical datasets and a real-world deprivation index.

Main Methods:

• Development of a greedy algorithm for ensemble model stacking.
• Comparison against linear stacking, genetic algorithm stacking, and brute force approaches.
• Application to optimize weighting of socio-economic factors for the German Index of Multiple Deprivation (GIMD) in relation to mortality.

Main Results:

• The greedy stacking algorithm achieved competitive ensemble weights, outperforming linear stacking in numerous tasks.
• While slightly less accurate than brute force, the greedy approach is significantly faster and more computationally efficient.
• The algorithm demonstrated broad applicability and efficiency, with a Python implementation available.

Conclusions:

• The proposed greedy stacking algorithm offers an effective and efficient solution for model ensembling.
• It provides a practical alternative to computationally intensive methods, suitable for various prediction tasks.
• The algorithm's flexibility is highlighted by its successful application in optimizing the GIMD for mortality correlation.