Toward Better Accuracy-Efficiency Trade-Offs: Divide and Co-Training.

Area of Science:

• Artificial Intelligence
• Machine Learning
• Deep Learning

Background:

• Neural network width increases model capacity but performance saturates.
• Wider networks do not always yield linear performance gains.
• Current scaling methods focus on width, neglecting ensemble approaches.

Purpose of the Study:

• To investigate ensemble methods as an alternative to increasing neural network width.
• To demonstrate improved accuracy-efficiency trade-offs using multiple smaller networks.
• To establish the number of networks as a new dimension for model scaling.

Main Methods:

• Dividing a large neural network into smaller ensemble members.
• Training small networks collaboratively with diverse data views (co-training).
• Validating the approach across 8 architectures on common benchmarks.

Main Results:

• Ensemble of small networks achieved superior performance compared to a single large network.
• Achieved better accuracy-efficiency trade-offs with comparable or fewer parameters and FLOPs.
• Demonstrated faster inference speeds through concurrent execution of small networks.

Conclusions:

• Ensemble methods offer a viable alternative to increasing neural network width for improved performance.
• The number of networks is a significant, underexplored dimension for model scaling.
• This strategy enhances both accuracy and computational efficiency in deep learning models.