Structured Ensembles: An approach to reduce the memory footprint of ensemble methods

Area of Science:

• Artificial Intelligence
• Machine Learning
• Deep Neural Networks

Background:

• Deep neural networks (DNNs) require substantial memory, limiting their deployment on resource-constrained devices.
• Existing ensembling methods often increase memory footprints, posing scalability challenges.
• Effective handling of continual learning tasks with limited memory remains an open problem.

Purpose of the Study:

• To propose a novel ensembling technique for DNNs that drastically reduces memory requirements.
• To develop a method for extracting diverse sub-networks from a single untrained network.
• To evaluate the proposed method's performance in terms of accuracy, memory efficiency, predictive calibration, and uncertainty estimation, and its applicability to continual learning.

Main Methods:

• A novel ensembling technique called Structured Ensemble is proposed.
• The method involves extracting multiple sub-networks from a single untrained DNN via end-to-end optimization.
• This optimization combines differentiable scaling with regularization terms to promote ensemble diversity.

Main Results:

• Structured Ensemble achieves higher or comparable accuracy to competing methods with significantly less storage.
• The ensembles demonstrate favorable predictive calibration and uncertainty estimation compared to state-of-the-art approaches.
• A modified framework effectively handles continual learning tasks with sub-linear memory cost, outperforming alternative strategies in average accuracy and memory usage.

Conclusions:

• Structured Ensemble offers a memory-efficient and effective approach to DNN ensembling.
• The method provides competitive accuracy and improved predictive uncertainty quantification.
• The framework shows promise for memory-constrained continual learning scenarios, mitigating catastrophic forgetting.