Online ensemble model compression for nonstationary data stream learning.

Area of Science:

• Machine Learning
• Artificial Intelligence
• Data Science

Background:

• Learning from data streams in nonstationary environments presents challenges due to evolving data distributions (concept drift).
• Traditional ensemble methods, while effective, incur high computational costs, limiting their use in resource-constrained applications.
• Existing approaches struggle to efficiently adapt to concept drift in real-time data streams.

Purpose of the Study:

• To introduce Online Weight Averaging (OWA), a novel online model compression technique for nonstationary data streams.
• To enable continuous compression of evolving neural network ensembles into a single, efficient model.
• To address the computational and memory constraints associated with traditional ensemble methods in dynamic environments.

Main Methods:

• Developed Online Weight Averaging (OWA), a stochastic weight averaging method for neural networks.
• Implemented a continuous compression strategy by averaging network weights at specific time steps.
• Incorporated a mechanism to forget outdated weights during concept drift events.

Main Results:

• OWA successfully compresses evolving neural network ensembles into a single model over time.
• The method demonstrates significant savings in computational cost compared to state-of-the-art ensemble methods.
• OWA achieves predictive performance comparable to existing ensemble techniques.

Conclusions:

• Online Weight Averaging (OWA) provides a robust and fast solution for learning from nonstationary data streams.
• OWA effectively leverages ensemble power within a single, computationally efficient model.
• The proposed method is suitable for applications with strict time and space requirements facing concept drift.