Incremental Trainable Parameter Selection in Deep Neural Networks.

Area of Science:

• Machine Learning
• Deep Learning
• Artificial Intelligence

Background:

• Deep learning models often require regularization to prevent overfitting and improve generalization.
• Stochastic Gradient Descent (SGD) is a common optimization algorithm for training deep learning models.
• The concept of effective degree-of-freedom (DoF) offers a way to quantify model complexity.

Purpose of the Study:

• To explore the utilization of a deep learning model's effective degree-of-freedom (DoF) for regularization during SGD-based training.
• To introduce a novel algorithm, incremental trainable parameter selection (ITPS), for governing the effective DoF.
• To demonstrate ITPS as a task and data-dependent regularization mechanism.

Main Methods:

• The incremental trainable parameter selection (ITPS) algorithm is proposed as a wrapper over SGD.
• ITPS incrementally selects parameters most sensitive to the training loss for updation.
• This process gradually increases the model's effective DoF during training.

Main Results:

• The ITPS algorithm demonstrated regularization-like behavior through gradual DoF increment.
• The algorithm achieved an optimum model configuration (DoF) for specific tasks.
• ITPS showed general utility across various neural network architectures (CNNs, transformers, RNNs, MLPs).

Conclusions:

• The ITPS algorithm provides an effective, data-dependent regularization strategy for deep learning.
• Gradually increasing the effective DoF of a model can enhance its performance and generalization.
• ITPS is applicable to diverse deep learning tasks, including image classification and healthcare.