Optimal training of integer-valued neural networks with mixed integer programming

Area of Science:

• Artificial Intelligence
• Machine Learning
• Optimization

Background:

• Current neural network (NN) training relies heavily on gradient-based methods, demanding substantial data, GPU computation, and hyper-parameter tuning.
• Training NNs using Mixed Integer Programming (MIP) solvers is an emerging area, offering potential advantages like reduced GPU and hyper-parameter tuning needs.
• Existing MIP-based NN training methods are limited by their capacity to handle only small datasets.

Purpose of the Study:

• To explore and advance the under-researched approach of training neural networks (NNs) using Mixed Integer Programming (MIP) solvers.
• To develop novel MIP formulations that enhance training efficiency and enable the training of integer-valued neural networks (INNs).
• To address the data limitations of current MIP-based NN training methods and optimize NN architecture during the training process.

Main Methods:

• Formulation of new Mixed Integer Programming (MIP) models for training neural networks (NNs), specifically targeting binarized NNs and integer-valued neural networks (INNs).
• Introduction of a novel method to optimize the number of neurons within the NN during the training phase, reducing reliance on pre-defined architectures.
• Development of a batch training approach to significantly increase the volume of training data that MIP solvers can effectively process.

Main Results:

• The proposed MIP models demonstrate improved training efficiency compared to previous methods.
• The new methods successfully train integer-valued neural networks (INNs) and optimize network architecture dynamically.
• Experimental results on two real-world, data-limited datasets show superior performance in accuracy, training time, and data utilization compared to the state-of-the-art in MIP-based NN training.

Conclusions:

• This research presents a significant advancement in training neural networks (NNs) using Mixed Integer Programming (MIP), particularly for data-limited scenarios.
• The methodology is proficient in training NNs with minimal data and low memory requirements, making it suitable for resource-constrained deployments.
• The developed techniques offer a promising direction for leveraging MIP solvers in NN training, overcoming previous scalability and efficiency challenges.