Structure-informed Risk Minimization for Robust Ensemble Learning

Area of Science:

• Machine Learning
• Artificial Intelligence
• Computer Science

Background:

• Ensemble learning enhances generalization but struggles with distribution shifts.
• Current methods optimize weights on validation data, failing in out-of-distribution (OoD) scenarios.
• Out-of-distribution generalization remains a critical challenge in machine learning.

Purpose of the Study:

• To develop a principled framework for learning robust ensemble weights.
• To improve out-of-distribution generalization without access to test data.
• To mitigate the limitations of existing ensemble combination strategies.

Main Methods:

• Proposed Structure-informed Risk Minimization (SRM) framework inspired by Distributionally Robust Optimization (DRO).
• Incorporated structural information of training distributions into uncertainty sets.
• Developed a computationally efficient optimization algorithm with theoretical guarantees.

Main Results:

• SRM learns robust ensemble weights by considering plausible real-world distribution shifts.
• The approach avoids the over-pessimism often associated with worst-case optimization.
• Demonstrated superior OoD generalization compared to existing methods across diverse benchmarks.

Conclusions:

• SRM offers a principled and effective approach to enhance ensemble robustness under distribution shifts.
• The framework provides a practical solution for improving generalization in unseen data distributions.
• SRM represents a significant advancement in out-of-distribution generalization for ensemble learning.